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ABSTRACT 

^The effects of a test coaching program, "Scoring High 
on the California Achievement Test , " were investigated with a sample 
of 876 students in grades 1, 2, 4^ and 5; Multivariate analyses of 
variance were used to determine the effects of the program, grade 
levels sex, and race. Sigiiif icaritdif f erences in favor of the Scoring 
High program were found on some of the subtest scores 6£ the 
California Achievement Tests (CAT) . Significant interaction effects 
prevented a straightforward interpretation of the program's impact. 
Some of the students in some grades increased_sdme CAT subtest scores 
as a result of participating in the program. The authors state that 
since the program did not produce clear evidence thatit is a 
valuable tool to increase student performance on the CAT, school 
administrators and others involved in the decision to use this 
particular program should weigh the results of this study as well as 
the financial requirements (approximately $2 .GO per student ) and the 
use of valuable classroom instruction time as they consider whether 
some small increases in some test scores for some students warrants 
the purchase and use of the program. (Author/PN) 
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The effect, of the Scoring High on the CAT program were 
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5- Mulcivariate analyses of variance were used to determine the 
effects of the program, .grade ievei, sex.and race. Significant 
differences (o<.01) in favor of the Scoring High program were 
found on some of the subtest scores of the CAT. Significant 
interaction effects prevented a straightforward interpretation of 
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THE EFFECTS OF A PROGRAM TO INCREASE CAT SCORES 

Considerable research and controversy have centered around the 
effects of programs designed to develop test-taking skills for 
aptitude and entrance tests (DerSimonian & Laird, 1983; Jackson , 
1980; Messick & Jungeblut, 1981; Slack & Porter, 1980). Although 
these effects are dependent upon the amount and kind of coaching, 
it has been concluded that scores oh aptitude and entrance measures 
can be influenced by coaching programs (Messick & Jungeblut, 1931; 
Slack & Porter, 1980) . But what about achievement tests? Are 
achievement test results also affected by programs designed to 
improve examinee performance? 

In 1954, Vernon hypothesized that coaching effects would be as 
great for achievement tests as they were for aptitude measures, but 
no empirical support for this statement was offered. Since that 
time, some studies regarding the effect of coaching on achievement 
tests have been done, but they have not received the attention of 
researchers as have aptitude tests chiefly because achievement test 
scores have tended not to be as crucial as aptitude test scores in 
many important scholastic decisions. 

To summarize the research conducted in this area, Bangert- 
Drowns, Kulik and Kulik (1983) performed a meta-analysis of studies 
of the effect's of coaching programs bri achievement test scores; 
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they considered interventions at three basic levels: (a) short 
practice sessions including a test- taking orientation * (bj longer 
interventions that included drill or practice on sample test items* 
and (c) work on broader cognitive skills (cf. Anastasi, 1981; 
Mehrens & Lehmann, 1984). While the individual investigations 
showed variations in effectiveness, the authors concluded that the 
typical coaching program did improve achievement test scores. They 
further found that the effects were smallest for short test-taking 
orientation sessions, larger ,in longer programs actually involving 
intensive drill arid practice, and largest in lengthy, broad skill 
training programs (cf. Mcphail, 1977). 

Studies reviewed by Banger t-Drowns et al . (1983) were limited 
and results were applicable only to the achievement tests and 
intervention programs included in their analyses. Much more 
research is needed before educators can use with confidence or 
riot use at all — the various intervention strategies that are 
available including the extensive commercially prepared programs. 
One such program in need of research is Scoring High on the 
California Achievement Test (Bates, 1981) . The publishers of 
advertise it as "America's best selling test preparation series 
with over 5,000,000 copies in print." Although this program is 
presented as if it will have a substantial impact upon student 
achievement on the California Achievement Tests, rib evidence is 
given to support this claim. This experimental investigation was 
undertaken to investigate the publisher's assertion. 
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Specifically, two questions were considered: 

1; What are the effects of the Scoring High program (Bates, 
1981) on scores obtained on the California Achievement Tests (CAT) 
for students in grades 1, 2, A, and 5? 

2. Are these effects moderated by grade level, sex, and race 
of the students? 

Procedures 

Participants in this investigation were 925 students in 40 
classes at five elementary schools within one city system in the 
Southeast. At grades 1, 2, 4, and 5 within each of the five 
elementary schools, two intact classes were randomly selected and 
randomly assigned to either an experimental group or to a control 
group. Altogether there were 460 students in the 20 classes 
comprising the experimental group and 465 in the 20 classes 
assigned to the control group. The experimental group received the 
Scoring High on the California Achievement Test program. The 
control group received no formal instruction in test-taking skills. 

The policy of the school system in Which this study was 
conducted was to stratify students by race (black/white) and sex at 
each grade level within each school and then assign students to 
classes alphabetically within each race and sex combination. While 
this process was not one of true randomization, it is reasonable to 
assume that the students in different classes within schools at 
each grade level did rtbt differ systematically. 



Scoring High oil the California Achievement Test is described by 
the publishers (Random House, 1983) as a program that enables 
students to master test-taking skills and strategies needed to 
maximize their performance on the California Achievement Tests. 
There are three books of the Scoring High program at three levels: 
Book A is for students in grades 1-4, Book B is for students in 
grades 3-6, and Book C is designed for students in grades 5-8. 
in this study, Book A was used for students in grades 1 and 2 and 
Book B was used with students in grades 4 arid 5. Each of the 30 
plus lessons in each book contained sample directions similar to 
those on the CAT and one or more sample items along with tips on 
test-taking strategies. Also included in each daily lesson was a 
practice section which included test items presented in formats 
similar to those on the CAT. 

The Scoring High program was completed by students in the 
experimental group in March and early April of 1983. Toward the 
end of April, both the experimental and control group students 
received the California Achievement Test battery as a part of the 
state test ing program. 

Criterion measures in the study were the scale scores attained 
by the students bri the Mathematics, Reading, and Language subtests 
of the CAT. Since the CAT subtest scores are not Comparable 
measures, no attempt was made to make comparisons across subtest 



scores . 
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Results and Concluaians 

Separate multivariate analyses of variance (MANOVA) were 
performed for grades 1 and 2 and for grades 4 and 5 since Book A of 
the Scoring High program was used in the first two grades and Book 
B was used in grades 4 and 5. Within each of these two grade level 
groups , MAN OVA s were conducted on each of the three sections of the 
CAT: Reading (vocabulary, comprehension, and total reading scores), 
Mathematics (computation , concepts and applications, and total 
mathematics scores) , arid Language (mechanics, expression, and total 
language scores) . A2X2X2X2 sampling design (group by grade 
by sex by race) was used to determine if any statistically 
significant differences existed between the groups or the 
interactions of the group effect with the other effets* 

The observed means and standard deviations for levels of main 
effects for each of the subtests of the CAT are shown in Table 1 
(students in grade 1 did not receive the CAT language subtests) . 
The students receiving the Scoring High program scored higher than 
the control group students oh ail CAT subtests except reading 
comprehension . Differences in favor of the experimental group 
varied from 5 to 14 scale score points which would correspond to 
about 2 or 3 months at most in grade equivalent scores. Table 1 
also shows differences on subtest scores between boys and girls, 
whites and blacks, and grade level. 

Tables 2, 3, and 4 contain the results of the multivariate 
analyses of variance for students in grades 4 and 5. 
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Statistically significant differences (p < -02 j wire found for 
three of the main effects (group, race, and grade) on all sets of 
CAT scores. A significant sex effect (p < .01) was noted tor the 
mathematics and language scores. Three statistically significant 
(P < .05) interaction effects were found but only one of these 
contained the group effect. The mathematics scores showed a 
significant multivariate F-statistir for the group X grade 
interaction effect. 

Since this study was designed to investigate the effects of the 
Scoring tfigh program, only those significant multivariate effects 
involving the experimental and control groups were followed with 
univariate analyses of varince (ANOVA) . Table 5 is a summary of 
the ANOVAs conducted on the CAT subtest scores which had 
significant multivariate group effects. 

The group X grade effect on the mathematics section of the CAT 
was attributed to control group students in grade 4 showing higher 
scores on all three mathematics scores than students in the 
experimental group. However, at grade 5, students receiving the 
Scoring High program outperformed the control group students on all 
mathematics subtests. Table 6 contains the meins and standard 
deviations of the CAT mathematics scores for the four combinations 
of the group by grade interaction effect. 

The significant MANOVA F statistic f6r the group effect on the 
three reading subtest scores Was followed with three univariate 
analyses of variance. From Table 5, though hone of the F-ratios 
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are significant at the .05 level > reading comprehension and reading 
vocabulary showed greatest differences between the experimental and 
control groups . The experimental group students had higher scores 
on vocabulary and slightly lower scores on comprehension than 
students in the control group. 

Language subtest scores showed differences between the groups 
on language mechanics and total language scores. The experimental 
group scored higher than the control group on both of these 
subtests (p< .01) . 

For students in grades 4 and 5, Scoring High on the CAT 
appeared to be somewhat, effective in raising scores on a few 
subtests in reading and language of the CAT. Mathematics scores 
for fifth grade students showed an increase but lower math subtest 
scores were observed for fourth grade students receiving the 
Scoring High program. 

Summaries of the MANOVAs performed on reading and mathematics 
subtest scores fcr students in grades 1 and 2 are given in Tables 7 
and 8. Since students in the first grade did hot receive the 
language portion of the CAT, Table 9 contains the MANOVA summary oh 
these subtest for students in the second grade. 

A significant (p< .05) four-way interaction was found on the 
reriihg subtest scores . Separate ANOVAs conducted on the three 
reading subtest scores suggested that only the reading vocabulary 
scores were involved in this interaction effect (see Table 10). 
uell means and standard deviations for reading vocabulary scores 
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corresponding to this interaction effect are shown in Table ii . 
Though hot conclusive, Table 11 reflects the very slight group 
differences at grade 1 as well as consistent sex differences for 
the white students at grade l. First grade black males showed 
higher scores than first grade black females in the experimental 
group but the reverse was true in the control group. 

Grade two results demonstrated little differences between the 
control and experimental groups for white students yet black 
females seemed to benefit by the Scoring High program even though 
black males did not. Also contained in Table 11 is the 
differential grade effect; white males and females and black males 
tended to show increases of about 40 or 60 scale score points from 
grade I to grade 2 regardless of the group to Which they had been 
assigned. Black females in the control group showed similar 
differences from grade 1 to grade 2 but black females who had 
received the Scoring High program in grade 2 were over 80 scale 
score units above the first grade black females in the experimental 
group. 

The MANOVA results on the mathematics subtest scores are 
presented in Table 8. A significant three-way interaction effect 
(p< .05) was found and followed with ANOVAs^ reported in table 10, 
determined that the group x sex X race interaction was only for the 
math computation subtest scores. Table 12 contains the means and 
standard deviation for the combinations of this interaction effect . 
White male and female students showed little differences in mean 
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scores between the control and experimental groups. However* black 
students in the Scoring high program had higher scores than black 
students in the control group; this difference was most pronounced 
for black males. Although differences appear to be small , the 
values in Table 12 are mean differences across both grade levels. 

Students in grade 2 did riot show significant multivariate 
effects involving the group variable on language subtest scores. 
Table 9 presents these results. 

The effectiveness of the Scoring High program on student 
achievement at grades 1 and 2 was difficult to interpret because of 
the confounding influence of sex* rac<; f and grade level on the 
group variable. The significant interaction effects may have been 
due, in part, to the level of the materials used. That is, Book A 
of the Scoring High program may not have been appropriate for these 
students. The differential effects observed may also have been 
related to a lack of test taking experiences of the students. 

In summary, Scoring High on the CAT did not produce clear 
evidence that it is a valuable tool to increase student performance 
on the CAT. School administrators and others involved in the 
decision to use this particular program should weigh the results of 
this study as well as the financial requirements (approximately 
$2.00 per student) arid the use of valuable classroom instruction 
time as they consider whether some small increases in some test 
scores for some students warrants the purchase and use of the 
program . 
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Table 1 

Means and Standard Deviations on CAT Subtest Scores 
for Levels of the Main Effects 



Read irig 



Vdcab Compr Total Mech 



Language 



Ma thema ti 



Exp 



Total Compu Gonce Total 



X 


433. 


5 


446. 


0 


431. 


9 


542 


.6 


503 . 


6 


490 


.2 


s 


75. 


9 


77. 


4 


80. 


0 


70 


.3 


70. 


2 


91 


.9 


N 


450 




447 




445 




337 




337 




337 




X 


428. 


8 


446. 


0 


429. 


2 


528 


c5 


498. 


2 


474 


.2 


s 


73. 


1 


84. 


1 


85. 


7 


57 


.9 


68. 


2 


86 


.5 


N 


465 




447 




462 




346 




345 




345 





Sex 





X 


428.5 


442.6 


427.4 


529.9 


498.8 


M 


S 


74.8 


83.7 


84.6 


67.2 


69 .6 




N 


476 


472 


471 


344 


343 




X 


433.9 


449.6 


434.0 


541 .0 


502.9 


F 


S 


74 .1 


77; 6 


81.0 


61.4 


68.8 




N 


439 


438 


436 


339 


339 


Race 
















X 


395.8 


405.2 


386.6 


504.1 


455. 1 


B 


S 


68.6 


68 .9 


74. 2 


62.1 


55.4 




N 


212 


212 


210 


148 


148 


W 


X 


442.2 


458 .9 


444.3 


544 .3 


513.9 


s 


72.8 


80.0 


80.6 


62.5 


66.8 




N 


700 


695 


694 


534 


533 



Grade 



475.7 
90.8 
343 

489.0 
87.4 

339 



432.3 
80 .3 
148 

497.6 
86.2 
533 



X 


345.0 


359.8 


335 


.6 














s 


31.2 


39.2 


38 


.4 














N 


230 


229 


227 
















X 


402.8 


415.0 


404 . 


.7 


478 


.5 


437 


.3 


455 


.0 


s 


27.8 


38.4 


43. 


4 


36 


.9 


39 


.4 


49 


.0 


N 


230 


226 


226 




226 




226 




226 


X 


465.5 


475.8 


463. 


3 


547 


. 2 


513 


.3 


525 


.4 


s 


46.2 


55.6 


53. 


1 


56 


. 1 


57 


.4 


62 


.4 


N 


219 


219 


218 




219 




218 




218 




X 


510 .6 


531 .6 


516. 


3 


578, 


.6 


54 9. 


.4 


561 


.9 


s 


49.3 


56.2 


54 . 




52. 


.3 


54. 


.5 


57. 


.2 


N 


236 


236 


236 




238 




238 




238 





403-6 438.2 
94 .7 75.2 
450 450 

397.2 423 .1 
91.6 66.0 
465 464 



324.9 429.6 
91.8 72.8 
472 472 

406.2 431.5 
94.2 69.3 
443 442 



371.5 393.4 
91 .0 64.6 

214 215 

409 .7 442.4 
91.8 68.8 
698 696 



424.4 
78;8 
449 

413.2 
71.3 
464 



416.3 
75.7 
471 

421.4 
74.8 
442 



387.0 
69 .0 
214 

428 .9 
74 .2 
696 



292.9 351.8 334.1 
29.6 36.8 27.9 

231 232 231 

349.9 405.5 382.6 
23.8 36.3 27.1 

227 226 226 

448.7 461.0 452.7 

46.4 43.8 41.3 

218 217 217 

508.0 502.9 503.9 

50.4 48.3 46.5 

239 239 239 



" « 13 

ERIC 



MANOVA Summary: 



Table 2 

Reading Vocabulary, Comprehension, Total 
Grades 4 and 5 



Source 



Wilk' s 
La mbd a 



Group 
Sex 
Race 
Grade 

Group X Sex 

Group X Race 

Group X Grade 

Sex X Race 

Sex X Grade 

Race x Grade 

Group x Sex X Race 

Group X Sex X Grade 

Group X Race x Grade 

Sex X Race x Grade 

Group X Sex X Race X Grade 

df = 3, 436 
**p < .01 
*p < .05 



.970 
.992 
.816 
.727 
.983 
.991 
.985 
.981 
.972 
.99 4 
.99 3 
.99 2 
.996 
.992 
.999 



4 .53** 

i.ii 

32.80** 
54.50** 

2.46 

1.34 

2.1- 

2.75* 

4 .25** 

0 .85 

1 .06 

1.16 

0.51 

1.19 

0.08 
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table 3 

MftNOVA Summary: Mathematics Computation, Concepts, Total 

Grades 4 and 5 



Source- 



Race 
Grade 



Wilk» s 
E ambda 



s|° UP • 927 11.54** 



966 5.07** 
.851 25.64** 



Group x Sex '™\ ^if* 

Group x Race "f f J ^.39 

Group X Grade .'HI 

Sex x Race - . 996 J:Jf 

Sex x Grade :||| J:" 

Race x Grade 994 

Group X Sex X RaCe " 998 a'ti 

Group x Sex X Grade " 991 

Group x Race x Grade "qoo 

Sex x Race X Grade " 99 8 f ~ m %\ 

Group x Sex X Race X Grade " 999 



df = 3, 438 
**p < .01 
*p < .05 
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Table 4 

MANOVA Summary: Language Mechanics , Expression , Total 

Grades 4 and 5 



Wiik' s 

Source Lambda F 



Gr°up .935 10.18** 

Sex -970 4;46** 

Race .813 33.69** 

Grade .809 34.48** 

Group X Sex .990 1.41 

Group X Race - 984 2.29 

Group X Grade .990 1.42 

Sex X Race .998 0.28 

Sex X Grade .986 2.11 

Race X Grade _ .996 0.64 

Group X Sex X Race .998 0.22 

Group X Sex X Grade .984 2.34 

Group X Race X Grade .996 0.58 

Sex X Race X Grade .989 1.64 

Group X Sex X Race X Grade .997 0.39 



df = 3, 438 
**p < .01 
*p < .135 



Table 5 

Summary of Analyses of Variance Involving Group Effect 

Grades 4 and 5 



df Variable MS Hyp MS Error F 



Group 1.438 Read Vocab 3278.05 1958.23 1.67 

Read Compr 6483.62 2637.66 2.46 

Read Total 969.02 2419.10 0.40 



Grou P 1,440 Math Compu 966.29 2189.22 0.44 

Math Conce 41727.95 1765 .91 23.63** 

Math Total 17265.30 1699.78 10,16** 

Group 1,440 Lang Mech 46196 .06 2608.48 17.71** 

Lang Expr 597.08 2606.36 0.23 

Lang Total 20684.75 3008.96 6.87** 

Group X Grade 1,440 Math Compu 35075.18 2189.22 16.02** 

Math Conce 13460.99 1765.91 7.62** 

Math Total 23231.70 1699.78 13.67** 



*p < .01 
*p < .05 
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Table .6 

Cell Means and Standard Deviations of the 
Group x Grade Effect on Mathematics Subtest Scores 

Grades 4 and 5 



Math Computation Subtest 
GrbUp Grade N x 



Control 4 111 457.02 39.86 

5 117 498.70 50.68 

Experimental 4 107 440.16 48.09 

5 122 516.92 50.06 

Math Concepts Subtest 

Control 4 110 457.44 31.90 

5 117 488.32 42.56 

Experimental 4 118 421.37 46.16 

5 122 516.88 45.82 

Math Total 

Control 4 110 454.32 31.02 

5 117 491.32 42.78 

Experimental 4 118 4C9.06 44.15 

5 122 516.03 ' 45.38 
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Table 7 

MANOVA Summary: Reading Vocabulary, Comprehension, total 

Grades 1 and 2 



Wilk's 

So urce Lambda 

Group .996 
Sex 



Race 
Grade 

Group x Sex ^995 



Group x Race .993 
Group x Grade [998 
Sex X Race !998 



df = 3, 432 
**p < .01 
*p < .05 



0.61 

.981 2.72* 



.780 40 .54** 

•452 174.65** 



0.69 
1.02 

998 0.33 

0.24 

999 0.17 



2.04 



Sex X Grade 

Race X Grade .986 

Group X Sex X Race .'990 ViB 

Group X Sex X Grade "939 

Group x Race X Grade ."999 0 

Sex X Race X Grade !991 1*26 

Group X Sex X Race X Grade ."980 2 !88* 
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Table 8 

MANOVA Summary: Mathematics Computation , Concepts^ total 

Grades 1 and 2 



^Source 



Wilkes 
Lambda 



J rou P -980 2.96* 

Se *_ .982 2,60 

* ac % -799 36.53** 

205.78** 



Grade .414 

Group X Sex .989 1.63 

Group x Race >982 2 64 

Group x Grade ,997 0*44 

Sex X Race .993 i] 02 

Sex X Grade ♦ .985 2.20 

Race x Grade .976 3.53* 

Group X Sex X Race .979 3,12* 

Group X Sex X Grade .990 1,52 

Group X Race X Grade i992 1^09 

Sex X Race X Grade .983 2*46 

Group x Sex X Race X Grade .992 1^22 



df = 3, 436 
**p < .01 
*P < .05 
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MANQVft Summary: 



Table-g 

Language Mechanics, Expression, Total 
Grade 2 



Source 



Group 

Sex 

Race 

Group 

Group 

Sex X 

Group 



X Sex 
X Race 
Race 

X Sex X Race 



Wiljc's 




Lambda 




.976 


1.75 


.934 


5.05** 


.728 


26.79** 


.978 


1.57 


.977 


1.69 


.930 


5.35** 


.973 


1.97 



df = 3, 215 
**p < .01 
*p < .05 
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table 10 

Summary of Analyses of Variance involving Group Effect 

Grades 1 and 2 



Source 



df 



Group 



Group X Race 



Group X Sex 
X Race 



X Sex 
X Race X Grade 



1*438 



1,433 



1,438 



1,438 



Variable 

Math Compu 
Math Conce 
Math Total 

Math Compu 
Math Conce 
Math Total 



^MS^yR MS Error 



Math Compu 
Math conce 
Math Total 



Read Voc 
Read Compr 
Read Total 



1579 . 10 
6541 .93 
2675.10 

4345 .57 
1336.15 
2239 .25 



4103.34 
744 .97 
1190.22 



4488 . 00 
2041.54 
1608. 64 



605. 25 
1085.82 
622.43 

*05.25 
1035.82 
622.43 



605.25 
1085.82 
522.43 



684 .25 
1297.50 
1375.89 



2.61 
6.02* 
4 .30* 

7. 18** 

1.23 

3.60 



6.78** 

0.69 

1.91 



6.56** 

1.57 

1.17 



**p < .01 
*p < .05 
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Table 11 

Means and Standard Deviations of Readiriq Vocabulary Scores 
Group X Sex X Race X Grade Effect 
Grades 1 and 2 



Group gex 



Male 
Male 

Control 

Female 
Female 



Grade 1 



R aae n 

White 44 

Black 16 

White 38 

Black 17 



Mean s _ 

352.6 27.7 

318.5 28.2 

355.4 23.5 

325.3 25.7 



Experimental 



Male 
Male 

Female 
female 



White 
Black 

White 
Black 



48 
18 

33 
11 



353.1 
330 .2 

359.6 
317.6 



26. 4 
23*9 

25.0 
36.0 



Grade -2 



Control 



Male 
Male 

Female 
Female 



White 
Black 

White 
Black 



45 

12 

43 
17 



404.6 
380. 3 

410.2 
383.0 



27.0 
30.7 

20.1 
32.3 



Male White 

Male Black 

Experimental 

Female White 

Female Black 



45 411.8 23.6 

9 374.3 34.8 

35 406. 1 25.2 

19 401.5 27.5 




= 20 - 
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Table 12 

Means and _ S taricia rd Deviations of 
Ma t hematics _CbfripLi tat ion .Scores 
Group X Sex X Race Effect 
Grades 1 and 2 Combined 



Group 



Sex 



Race 



N 



Mean 



Control 



Male 
Male 

Female 
Female 



White 
Black 

White 
Black 



90 
29 

81 
35 



324*5 
286*6 



330*6 
308.2 



25.9 
27.3 

23.1 
26.0 



Male White 

Male Black 

Experimental 

Female White 

Female Black 



93 325.2 21.8 

27 313.6 31.1 

67 328.2 21.0 

32 315.5 29.0 
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